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Abstract Body 


Background / Context: 

Interest in sealing up prekindergarten programs for ehildren from low income families is 
dramatically increasing, especially with the announcement by President Obama of support for 
“universal preschool” (Scott, 2013). A concern with expansion is that quality— and the ensuing 
effects-is hard to maintain when a program is taken to scale (Baker, 2011). Many states 
currently use the Early Childhood Environment Rating Scale - Revised edition (ECERS-R; 
Harms, Clifford, & Cryer, 2005) in combination with other measures to evaluate the quality of 
state-funded early care programs. In several states, both the ECERS-R and the Early Eanguage 
and Eiteracy Classroom Observation (EEECO; Smith & Dickinson, 2002) must be completed if a 
classroom serves preschool-age children. In addition to the general quality that is the focus of 
measures like the ECERS-R and EEECO, classrooms differ widely in the time percentages 
allotted to different activities, with many classrooms spending an extremely large portion of the 
day in routine practices like bathroom breaks, meals, naptime, and transitions. Research in early 
education supports the idea of developmentally appropriate practice as reflected in instructional 
groupings and covered content (Bredekamp, 1987); however, it is not yet clear if such 
appropriate practice and consistency in that practice can be upheld in a statewide program. This 
paper focuses on evidence from one portion of the TN-VPK evaluation from which classroom 
observations yield information on classroom quality state-wide. 

Purpose / Objective / Research Question / Focus of Study: 

This paper focuses on the variability in program characteristics associated with TN-VPK 
classrooms observed across the state of Tennessee. 

Setting: 

As described in the previous papers’ abstracts, TN-VPK supports 934 pre-k classrooms that 
serve more than 18,000 at-risk children in 133 of the 136 Tennessee school systems. It is, 
therefore, truly a statewide program serving all areas of the state, both rural and urban. 

Population / Participants / Subjects: 

This paper includes a sample of 160 TN-VPK classrooms that were intentionally chosen to be 
representative of the distribution of TN-VPK program types across the state of Tennessee. 

Those classrooms were located in each of the state’s four regions: 40 in West Tennessee 
(including Memphis), 39 in Central West Tennessee (including Nashville), 40 in Central East 
Tennessee (including Chattanooga), and 41 in East Tennessee (including Knoxville). This study 
is a partnership with the Tennessee Office of Early Eearning (OEE) of the TN Department of 
Education. Administrators in OEE are responsible for the TN-VPK program. As the 
administrators in OEE requested, our sample of 160 included 43 classrooms that were part of the 
original pilot program that Tennessee began in 1998 and 117 classrooms that were put into place 
following that pilot period. The majority of the classrooms (118) were located inside of a public 
school, although the sample also included 35 classrooms off-site and 7 classrooms in locations 
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that served pre-k only. Per the requirements of the TN-VPK system in Tennessee, these 160 
elassrooms primarily served ehildren from low-ineome families. 

Intervention / Program / Practice: 

A full description of the TN-VPK program is given in the abstract of the first paper in this 
symposium proposal. 

Research Design: 

The two previous papers have focused on the randomized control trial portion of the larger study 
of the effectiveness of the TN-VPK program. A second major component of the evaluation 
project involves an age-cutoff regression discontinuity design (RDD). Two cohorts of children 
in a representative sample of 160 classrooms across the state of Tennessee are participating in 
this phase of the study. Individual assessments on the same literacy, language, and math 
measures used in the RCT are being administered at the beginning of kindergarten for the cohort 
of children who were enrolled in TN-VPK in one of these schools during the previous school 
year. The scores for those children will then be compared with scores obtained from the cohort 
of children just entering TN-VPK in those same schools that same school year. With the 
difference in age accounted for statistically, this comparison will provide another set of estimates 
of the effects of TN-VPK. Although to date complete assessment data have been collected from 
only two of four state regions, the data from all four regions will provide a representative sample 
of TN-VPK programs statewide. These data will provide the state the opportunity to investigate 
whether programmatic characteristics are linked to differential child outcomes. 

Unlike in the RCT portion of the evaluation, classrooms in the RDD portion are identified and all 
children in each classroom are assessed. Each classroom is observed for a full day. The OEL 
administrators identified “policy relevant” questions that they would like this portion of the full 
evaluation to address and observational measures were chosen accordingly. Eventually, the plan 
is to link observed differences among the classrooms to gains made by children in the 
classrooms. While we are not prepared to make those links in this paper, we are prepared to 
describe the differences found in program implementation, some of which are conceptually 
linked to the variation in effects obtained in the RCT sample of classrooms. 

Data Collection and Analysis: 

As part of the RDD, trained and reliable members of the PRI research team conducted in-depth 
systematic observations of the 160 pre-k classrooms. Using measures described below, we 
collected detailed information about the environments to which children who participate in TN- 
VPK are exposed. Observations took place on one day during the winter and lasted a minimum 
of four hours, covering the period of time prior to nap, typically considered the prime 
instructional time of the day in pre-k classrooms; 23 classrooms, spread evenly across the 
regions, were observed for the full day by two observers for inter-observer reliability. Measures 
included the following: 

• Early Childhood Environment Rating Scale - Revised Edition (ECERS-R). The ECERS- 
R is one of the most widely-used instruments to evaluate the quality of early care and 
education environments. It is comprised of 43 items grouped into 7 subscales. Many 
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studies using the ECERS-R, ineluding our study, use only the first six subseales (omitting 
the Parents and Staff subseale). The ECERS-R items are seored on a seale ranging from 
1 (representing Inadequate quality) to 7 (Exeellent quality). The total ECERS-R seore is 
the simple average of the six subseale seores. ECERS reliability was seored as exact 
agreement between observers. With this striet eriterion, observers aehieved 88% 
agreement (Kappa = .84). 

• Early Eanguage and Eiteraey Classroom Observation (EEECO). The EEECO is an 
instrument used to evaluate the language and literaey environments of early ehildhood 
elassrooms. The rating seale portion of the instrument involves 19 items grouped into 5 
subseales (whieh are then further eombined into 2 larger subseales and a total seale by 
averaging the small subseale seores). Eaeh of the 19 items is rated on a 5-point seale 
with 1 representing Defieient quality and 5 representing Exemplary quality. In addition 
to these 19 items, the EEECO ineludes a Eiteraey Environment Cheeklist whieh involves 
the ratings of presenee/absenee and quality of the physieal environment in the elassroom 
that pertains to language and literaey ineluding books, writing materials, and elassroom 
displays. EEECO reliability was seored as exact agreement between observers. With this 
striet eriterion, observers aehieved 83% agreement (Kappa = .72). 

• Narrative Reeord. The Narrative Reeord (Earran & Bilbrey, 2004) uses an open-ended 
format and eoding seheme to reeord eontinuous data about the progression of aetivities in 
the elassroom. It eonsists of observer notes and eodes for eaeh distinet episode that 
oeeurs, e.g., a small group aetivity. The data eolleeted deseribe, at a minimum depending 
on whieh variable eategories are ineluded, the timing of the episode (start and stop times), 
the learning setting, and the learning foeus. Beeause it is eontinuously reeorded, the 
Narrative Reeord eaptures the entire observational period with no breaks in eoding. The 
basie eodes used in the Narrative Reeord aeross all four regions of the state eovered 
primary divisions of aetivity setting used and instruetional eontent of foeus. Inter- 
observer agreement for the Narrative Reeord for 23 observation visits was defined by 
agreement within 3% of the total minutes observed in an episode type. Eor instanee, if 
Observer 1 reported 30% of the observation spent in meals and Observer 2 reported 32% 
of that same observation spent in meals, the observer pair was deemed to be in agreement 
for that variable eategory. In reliability visits in partieipating elassrooms, there was a 
mean observer agreement aeross the 1 1 Narrative Reeord aetivity setting eategories of 
94% (standard deviation of 7.3%). Eor eontent eategories, mean observer agreement 
aeross the 10 eategories was also 94% (standard deviation of 5.4%). 

Findings / Results: 

Table B1 presents deseriptive statisties on summary seores for the three observation measures. 
Mean seores indieate the programs were, on average, less than what is eonsidered “Good” 
quality aeeording to ECERS-R anehors (i.e., seore of “5”). On the EEECO, mean seores all 
hovered near the anehor representing “Basie” evidenee of quality. The Narrative Reeord showed 
that elassrooms in this sample spent an average of 40% of the first four hours of the day, or 95 
minutes, in routine aetivities ineluding meals and transitions. Ear more time was spent in whole- 
group aetivities (38%) than in any grouping involving eenter time (22%). What is more 
interesting than mean seores for the purposes of this paper, however, is the variation in seores 
aeross programs. Variation was present in all three elassroom observation measures. Eor 
instanee, within this sealed up prekindergarten program, there was one elassroom with an 


SREE Spring 2014 Conference Abstract Template 


3 



average ECERS seore of 2.51, a seore that the ECERS authors eonsider very elose to Inadequate 
and therefore dangerous. Yet there was another program with an average seore of 6.08, whieh is 
elose to the maximum seore on the seale and in the Exeellent range. Similarly, in terms of how 
time was spent in the elassroom, one elassroom was observed spending almost 3 of its 4 hour 
morning observation in non-learning activities (routines, transitions, meals) while another spent 
52 minutes of the 4 hours on those activities. Great variation was observed in how the day is 
structured and whether children are receiving most of their instruction in whole group teacher- 
led activities or spending some of their time in center-based, more independent activities. 

Conclusions: 

Our classroom observations of the statewide implementation of TN-VPK have demonstrated the 
same huge variability in program implementation that we saw in program effects in the RCT 
findings. Granted these are not the same classrooms and we have not yet examined the links 
between the observational variability obtained and child outcomes. However, at a larger level, 
we can conclude that statewide implementations of prekindergarten programs (for example, 
“Universal Pre-K”) may have a difficult time ensuring quality control. The TN-VPK program is 
not alone in pushing to enact a program statewide without a clear vision communicated to its 
program participants of what should be happening in the classrooms. Moreover, as with other 
scaled up programs, few funds were allocated at a central level to supervise the quality of the 
individual classrooms or to determine and create the kind of professional development that may 
be needed. The children these programs are meant to serve are among the most vulnerable. 
Determining what aspects of pre-k programs should be uniform in order to serve the children 
well is an important, overdue step. 
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Appendix B, Tables and Figures 

Table B1 

Descriptive Statistics for Observation Measures - Subscales and Total Scores 


Minimum Maximum Mean SD 


ECERS 

Space and Eurnishings 
Personal Care Routines 
Eanguage and Reasoning 
Activities 
Interactions 
Program Structure 
Total Score 
EEECO 

Classroom Structure 
Curriculum 

Language Environment 
Books and Book Reading 
Print and Early Writing 
Total Score 
Narrative Record 
Activity Setting 

% Time in Routines 
% Time in Whole-Group Instruction 
% Time in Instruction Involving 
Centers 

Content of Eocus 

% Time Eocused on Literacy 
% Time Eocused on Math or Science 
% Time Eocused on Mixed Content 
% Time Without Any Instructional 
Content 


2.63 

6.13 

4.23 

0.66 

1.80 

6.50 

3.79 

1.02 

1.75 

7.00 

4.64 

1.08 

1.56 

6.30 

3.15 

0.86 

2.00 

7.00 

5.39 

1.05 

1.33 

7.00 

3.99 

1.22 

2.51 

6.08 

4.20 

0.75 

2.25 

5.00 

3.81 

0.59 

1.67 

4.67 

3.26 

0.62 

2.00 

5.00 

3.37 

0.65 

1.60 

5.00 

3.46 

0.58 

1.33 

5.00 

3.14 

0.70 

1.88 

4.79 

3.43 

0.50 


0.22 

0.68 

0.40 

0.08 

0.10 

0.68 

0.38 

0.11 

0.00 

0.52 

0.22 

0.10 

0.00 

0.48 

0.16 

0.08 

0.00 

0.20 

0.04 

0.05 

0.04 

0.56 

0.29 

0.11 

0.20 

0.62 

0.38 

0.08 
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